Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix strange straggler issues #158

Merged
merged 6 commits into from
Feb 22, 2022
Merged

Fix strange straggler issues #158

merged 6 commits into from
Feb 22, 2022

Conversation

parasj
Copy link
Contributor

@parasj parasj commented Feb 22, 2022

This is a large PR that contains changes to address stragglers including:

  • Better retry logic for the post request to register a chunk request which fails occasionally
  • Improved disk space backoff which now applies to random chunk generation and uses df to efficiently query the filesystem superblock
  • Minor changes to pass pytype which fixes some minor bugs
  • tmpfs for the chunk store instead of using /dev/shm
  • Use loop around recv for chunk headers

@parasj parasj self-assigned this Feb 22, 2022
@parasj parasj added the gateway label Feb 22, 2022
@parasj parasj added this to the Skylark 0.1 alpha critical milestone Feb 22, 2022
@parasj parasj linked an issue Feb 22, 2022 that may be closed by this pull request
@parasj parasj merged commit 483b72c into main Feb 22, 2022
@parasj parasj deleted the dev/paras/debug_reset branch February 22, 2022 22:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug] Debug consistent appearance of stragglers
1 participant